Not a database file

nate.berkopec's Avatar

nate.berkopec

20 Jan, 2018 07:50 PM

When loading SARTopo Offline (nightly or stable) with *any* .mbtiles files in the tiles folder, I get a crash and stacktrace. With no tiles in data/tiles, the app boots fine. I've tried dozens of combinations of mbtiles, I can't find any particular ones that work, the app seems to just not boot with any of them. Wiping out the data/db.* files has no effect.

Is it possible all of my .mbtiles (~200GB worth or so) are corrupt? How can I check?

java.sql.SQLException: [SQLITE_NOTADB] File opened that is not a database file (file is encrypted or is not a database)
at org.sqlite.core.DB.newSQLException(DB.java:890)
at org.sqlite.core.DB.newSQLException(DB.java:901)
at org.sqlite.core.DB.throwex(DB.java:868)
at org.sqlite.core.NativeDB.prepare(Native Method)
at org.sqlite.core.DB.prepare(DB.java:211)
at org.sqlite.jdbc3.JDBC3Statement.executeQuery(JDBC3Statement.java:81)
at org.sarsoft.offline.FSLocalTileProvider.init(FSLocalTileProvider.java:76)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:344)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:295)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:396)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1507)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:191)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:921)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:864)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:779)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:490)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:87)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:287)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1148)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:191)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:921)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:864)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:779)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:490)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:87)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:287)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1148)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:191)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireByName(AbstractAutowireCapableBeanFactory.java:1178)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1128)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:191)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1119)
at org.springframework.web.servlet.handler.AbstractUrlHandlerMapping.registerHandler(AbstractUrlHandlerMapping.java:287)
at org.springframework.web.servlet.handler.AbstractUrlHandlerMapping.registerHandler(AbstractUrlHandlerMapping.java:266)
at org.springframework.web.servlet.handler.AbstractDetectingUrlHandlerMapping.detectHandlers(AbstractDetectingUrlHandlerMapping.java:82)
at org.springframework.web.servlet.handler.AbstractDetectingUrlHandlerMapping.initApplicationContext(AbstractDetectingUrlHandlerMapping.java:58)
at org.springframework.context.support.ApplicationObjectSupport.initApplicationContext(ApplicationObjectSupport.java:119)
at org.springframework.web.context.support.WebApplicationObjectSupport.initApplicationContext(WebApplicationObjectSupport.java:75)
at org.springframework.context.support.ApplicationObjectSupport.setApplicationContext(ApplicationObjectSupport.java:73)
at org.springframework.context.support.ApplicationContextAwareProcessor.invokeAwareInterfaces(ApplicationContextAwareProcessor.java:117)
at org.springframework.context.support.ApplicationContextAwareProcessor.postProcessBeforeInitialization(ApplicationContextAwareProcessor.java:92)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:396)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1507)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:191)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:636)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:934)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:410)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)
at winstone.WebAppConfiguration.<init>(WebAppConfiguration.java:889)
at winstone.HostConfiguration.initWebApp(HostConfiguration.java:131)
at winstone.HostConfiguration.<init>(HostConfiguration.java:73)
at winstone.HostGroup.initHost(HostGroup.java:85)
at winstone.HostGroup.<init>(HostGroup.java:45)
at winstone.Launcher.<init>(Launcher.java:196)
at winstone.Launcher.main(Launcher.java:391)
at sarsoft.Loader.main(Loader.java:146)

  1. Support Staff 1 Posted by matt on 20 Jan, 2018 09:11 PM

    matt's Avatar

    What OS are you running?

    I've never encountered this before and I don't have any immediate thoughts. This won't be a perfect test as the Java SQLite driver I'm using doesn't rely on any system-installed files, but if you download sqlite3 and run

    sqlite3 filename.mbtiles "select value from metadata where name='group'"

    does it throw the same error, or show a result? I doubt the query matters here but that's the query that the app is failing on.

  2. 2 Posted by nate.berkopec on 20 Jan, 2018 09:40 PM

    nate.berkopec's Avatar

    Thanks for the fast reply.

    Mac OS X. It does show a result, e.g "aspect" when I run it against an
    aspect file.

    As an example, the md5 checksum I have on aspect-36-108.mbtiles
    is e4a6f5f58223b8e8c0717e445794ddac. I can't check if that's correct
    because I exceeded my download allowance.

  3. Support Staff 3 Posted by matt on 20 Jan, 2018 10:08 PM

    matt's Avatar

    That's the correct md5 checksum. I'll need to do some research, don't have anything helpful to say at the moment.

    • Have you been able to run the app with .mbtiles before, or did this happen the first time you tried?

    • I gave you the ability to download some more data. Can you download a new .mbtiles file directly to the data/tiles directory, and then start the app? Pure speculation but I'm wondering if the existing have some kind of attribute set which is causing this.

  4. 4 Posted by nate.berkopec on 20 Jan, 2018 10:37 PM

    nate.berkopec's Avatar

    > Have you been able to run the app with .mbtiles before, or did this
    happen the first time you tried?

    Ran it a few times with tiles. It seems to have stopped working after I
    downloaded a *lot* of tiles (like 100-150GB) but like I said, moving tiles
    from data/tiles to data/not_tiles allows the app to start.

    > I gave you the ability to download some more data. Can you download a new
    .mbtiles file directly to the data/tiles directory, and then start the app?
    Pure speculation but I'm wondering if the existing have some kind of
    attribute set which is causing this.

    No dice, same stacktrace.

    I get the same behavior if I wipe out the install, then re-install from
    scratch and add in my old tiles folder, so I think you're right that
    something must be corrupt in the tiles themselves.

    It seems like the stacktrace gets printed for every tile in the folder
    (twice for two files, thrice for three, so the stack is really long with
    the whole tiles folder I have).

    If my .mbtiles have the same md5 as yours though, there's no way they could
    even be different, right?

  5. Support Staff 5 Posted by matt on 21 Jan, 2018 12:30 AM

    matt's Avatar

    I'm not familiar enough with SQLite's inner workings to really comment, but I was thinking it might be metadata (via the mac filesystem's extended attributes) rather than the file contents themselves. If you downloaded a fresh file, that seems less likely.

    What version of Java are you running (java -version)? I've tested on Java 9 but I'm not sure I've tested mbtiles files specifically and Java 9 introduced a number of issues for existing libraries.

    On the startup screen, you should see something like:

    Temp data unpacked to /var/folders/zg/vg69mzg13h75vkdgtcjdky5w0000gn/T/winstoneEmbeddedWAR

    Can you wipe that entire directory (your equivalent of the T directory), or since it includes non-SARTopo files as well, to be safe you can run the following on the command line instead:

    java -Djava.io.tmpdir=[specify a new temp directory] -Dsarsoft.properties=sar.properties -jar sar.jar

    I try to wipe the winstoneEmbeddedWAR directory on startup, but even if you delete and replace the app, the temp directory as a whole gets reused and the Java SQLite library might have unpacked something in there that's getting read on startup.

  6. 6 Posted by nate.berkopec on 21 Jan, 2018 05:06 AM

    nate.berkopec's Avatar

    Java 8 (java version "1.8.0_65")

    Unfortunately, changing the tmp directory to a new, fresh one didn't change
    anything either :(

  7. Support Staff 7 Posted by matt on 23 Jan, 2018 06:01 PM

    matt's Avatar

    I'm going to need a little more time to get back to you, but I also need to know how deep you want to dig on this - I'm happy to work on it until it's resolved, but with no idea what the root cause is, that's probably going to mean a lot of back and forth and trial and error.

    At this point the best next step I can think of is to load the app plus mbtiles onto a thumbdrive, try running it off the thumbdrive, and then try running the thumbdrive on a new computer. That would at least confirm that it's something cached outside of the app on your computer, and nothing within the sartopo offline file tree.

    If you don't have another ccmputer to try on, I can send you a thumbdrive with an install that I've already tested.

  8. 8 Posted by nate.berkopec on 23 Jan, 2018 06:20 PM

    nate.berkopec's Avatar

    No worries Matt. I'm a web developer too so I know how this goals.

    I've put the app and mbtiles on a thumbdrive and ran the app on a Windows
    computer. No dice, exact same behavior and stacktrace is printed.

  9. Support Staff 9 Posted by matt on 23 Jan, 2018 06:42 PM

    matt's Avatar

    That's actually really good news since it should be easier to debug, but I'm also really puzzled at how that pairs with the issue showing up after a fresh install.

    Can you send the directory listings (ls -l) for the main directory, data, and data/tiles? If nothing pops out, the next step will probably be to zip it up with a small mbtiles file inside and send me a copy (or do that via thumdrive if necessary). If it persists across machines, hopefully I can debug it myself without needing to go through you for everything.

  10. 10 Posted by nate.berkopec on 23 Jan, 2018 08:16 PM

    nate.berkopec's Avatar

    Here are the ls results:
    https://gist.github.com/nateberkopec/82be7f91aa1d69c569aea4e34f66d979

    An interesting development as I was preparing a zip for you: running a
    minimal repro case from my thumbdrive results in a stacktrace. I didn't
    mention it but I've solely been running SARTopo from an external USB 3 SSD.
    Moving the same files to my harddrive and voila, it works fine. So clearly
    I have a read issue of files from my hardware here.

  11. 11 Posted by nate.berkopec on 23 Jan, 2018 08:29 PM

    nate.berkopec's Avatar

    Man, this is really wacky. To summarize:

    * On my mac, running SARTopo + tiles from my external SSD (over USB 3.0) results in a stacktrace.
    * On my mac, running SARTopo + tiles from my internal SSD works fine (or at least doesnt spit a stacktrace, I'm not sure if the tiles actually work or not)
    * On my Windows PC, running SARtopo + tiles from the internal SSD *or* the external SSD results in a stacktrace.

    I have a repro .zip available at a URL, I won't post it here but can send the URL to you, just give me a email/etc.

  12. Support Staff 12 Posted by matt on 23 Jan, 2018 11:01 PM

    matt's Avatar

    You can send it to [email blocked], but I think I'm going to hold off for a minute.

    Is your external drive formatted as NTFS or FAT32? Running of an external drive is definitely supported but I've encountered issues in the past swapping external NTFS drives between machines, since the usernames don't line up. Since FAT doesn't have a concept of file owners, it hasn't had this issue. If that were the case, I'd expect a different error message, though.

    You can store the app and map data in different locations by editing sar.properties and setting

    sarsoft.map.localTileStore=/path/to/tiles/folder

    I'm assuming this will work with the app on the external drive and the localTileStore on your local machine, but not the other way around - is that the case?

  13. 13 Posted by nate.berkopec on 24 Jan, 2018 01:15 AM

    nate.berkopec's Avatar

    It's FAT formatted.

    Yup, your hunch is correct. Running with the exact same tiles folder
    containing one mbtile file, I can launch and run the app just fine if it's
    pointing to my internal drive with sar.properties, and I get the usual
    stacktrace when running straight off the drive.

  14. Support Staff 14 Posted by matt on 24 Jan, 2018 07:49 PM

    matt's Avatar

    I think I was able to reproduce this on my local environment.

    When copying files with extended attributes over to a FAT drive using either the command line or the finder, mac os creates hidden ._ files holding the extended attribute information. It also seamlessly deletes these hidden files when you delete the primary file. My app is picking those up as MBTiles files and trying to read them. I don't think there's anything you can do via the finder, but you can delete them via the command line.

    When copying the files onto an HFS+ drive, the ._* files automatically get collapsed back to extended attributes, which is why it worked locally.

    If that doesn't work, I have a development .jar file I can share with you that logs the problematic filenames rather than just throwing an exception.

  15. 15 Posted by nate.berkopec on 24 Jan, 2018 09:55 PM

    nate.berkopec's Avatar

    Ta-da! Looks like you were correct. Also, I guess dot_clean
    <https://lifehacker.com/377011/delete-mac-system-files-with-dotclean> is a
    thing.

    You can just put a fix in the next release that ignores mbtiles which start
    with ._, yeah?

    Until then, I'm happy to work around this, it's not a major issue.

    Also FWIW the "latest stable version" links at
    https://sartopo.com/offline/app still point to v3520, not 3786.

  16. Support Staff 16 Posted by matt on 25 Jan, 2018 07:14 PM

    matt's Avatar

    The latest stable version issue has been addressed with the new release.

    The error reporting will be less annoying when my fix goes live, but I'll try to ignore ._* files as well.

  17. matt closed this discussion on 25 Jan, 2018 07:14 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac