Getting Django SQLite file out of Web App

Question

Getting Django SQLite file out of Web App

Matthew Kestenbaum 20

I released my website a couple weeks ago, and having no experience with deployment before, I uploaded my Django files - including SQLite - to Github and used Github CI to deploy my Web App. This was never supposed to be a permanent solution, just something to get my site up and running. However, if I make any changes now to my local repository and commit them, it will undo all of the website database changes.

Luckily my website is just starting, and there is not that much new data in the SQLite database. However, there are 10s of users with data, and I don't want to get rid of it. I of course want to switch to a better and more robust database system. However, before I do this I need to retrieve the data from my SQLite database, that is within Azure (not my local one).

How can I extract my SQLite database from my Azure Web App? With FTP or Kurus, I am able to download a .tar.gz that has my sqlite file in it. However, this file is not updated with the server versions of the database. They only have the same one that is on Github. I know there are Django scripts that can extract from the database, and am thinking about that. But, in order to do this, I need to be able to upload this Django script to the Web App - which if done through Github CI, will revert my database.

I know my own ignorance got me into this issue, but is there any way out without getting rid of all my user data that isn't local ? Really appreciate any thoughts / help on this one.

ajkuma 28,036 Reputation points Microsoft Employee Moderator

2023-08-27T20:21:23.2866667+00:00

Apologies for the delay from over the weekend (previous thread), and long response' on this.

Based on your issue description, I understand you are looking to extract the data from your SQLite database that is hosted on your Azure Web App. You mentioned that you are able to download a .tar.gz file that contains your SQLite file, but it is not updated with the server versions of the database.

Typically, the SQLite database file is stored in a temporary directory on the server and is not persisted across deployments or server restarts. Instead, a new database file is created each time your web app starts up. To download the current version of your SQLite database file, you can try the following steps:

1.      Open the Kudu console for your web app. You can access the Kudu console by going to https://.scm.azurewebsites.net.

2.      Navigate to the \home\data directory.

3.      Look for a file with a .db or .sqlite extension. This is your SQLite database file.

4.      Download the file to your local machine using the Kudu console's download feature.

I understand you are considering to switch to a better and more robust database system. Just to highlight, the file system of your WebApp is a mounted network share. This enables scale out scenarios where your code needs to be executed across multiple hosts. Unfortunately this blocks the use of file-based database providers like SQLite since it's not possible to acquire exclusive locks on the database file. We recommend a managed database service: Azure SQL, Azure Database for MySQL or Azure Database for PostgreSQL.

Can I use a file based database (like SQLite) with my Linux Webapp?

Also, as mentioned in this doc - It isn't recommended to use storage mounts for local databases (such as SQLite) or for any other applications and components that rely on file handles and locks.

Based on your requirement, if feasible and works fine, you may persist your database, try to save your SQLite file inside /home/site/wwwroot folder as mentioned. Checkout this discussion thread.
Matthew Kestenbaum 20 Reputation points

2023-08-28T04:09:18.4233333+00:00

Hi @Bharathi G, Rajkumar , I'm a little confused by Kudu. If I access the File Manager, all I see is the same file system I see through FTP, which does not have a /home/data directory. Within wwwroot, the only files I see are "output.tar.gz", "oryx-manifest.toml", "hostingstart.html" and ".ostype". When I download 'output.tar.gz' and extract it, the db.sqlite3 file is the same as the original one from my Github repo - I.E it is not updated with changes made to the server version.

However, when I use SSH with Kudu, and a type 'ls', it lists my normal file Django file system, including the 'db.sqlite3'. However, I'm not sure how to download this from SSH. It is not clear, and I'm also not sure if this is the updated version or not. In neither scenario did I find a 'home/data' directory.

Not sure if it is relevant, but I am deploying using Github CI.
ajkuma 28,036 Reputation points Microsoft Employee Moderator

2023-09-01T03:04:44.0033333+00:00

@Matthew Kestenbaum , Just checking in to see if you had got a chance to see the previous response. If the answer helped (pointed, you in the right direction) > please click Accept Answer Or please share the requested/more info to help you better.

2 answers

Your answer

Matthew Kestenbaum 20 Reputation points

2023-08-28T04:09:18.4233333+00:00

Hi @Bharathi G, Rajkumar , I'm a little confused by Kudu. If I access the File Manager, all I see is the same file system I see through FTP, which does not have a /home/data directory. Within wwwroot, the only files I see are "output.tar.gz", "oryx-manifest.toml", "hostingstart.html" and ".ostype". When I download 'output.tar.gz' and extract it, the db.sqlite3 file is the same as the original one from my Github repo - I.E it is not updated with changes made to the server version.

However, when I use SSH with Kudu, and a type 'ls', it lists my normal file Django file system, including the 'db.sqlite3'. However, I'm not sure how to download this from SSH. It is not clear, and I'm also not sure if this is the updated version or not. In neither scenario did I find a 'home/data' directory.

Not sure if it is relevant, but I am deploying using Github CI.
ajkuma 28,036 Reputation points Microsoft Employee Moderator

2023-09-01T03:04:44.0033333+00:00

@Matthew Kestenbaum , Just checking in to see if you had got a chance to see the previous response. If the answer helped (pointed, you in the right direction) > please click Accept Answer Or please share the requested/more info to help you better.

Answer 1

@Bharathi G, Rajkumar I can't access the filesystem through 'Kudu -> Debug Console' because I'm running a Web App on Linux. When I try to access the file system through the "Browse Directory -> wwwroot" it shows the same files as when I FTP in, which "output.tar.gz", "oryx-manifest.toml", "hostingstart.html" and ".ostype". When I download and extract 'output.tar.gz' it gives me the SQLite file from my Github repo, not updated with the current server database.

When I use SSH in Kudu, and I type 'ls', it outputs me my Django file system including the SQLite file. However, I'm unsure how to download the file from SSH? It is just a console and does not have any Kudu file management system in it. Also, I'm not sure if this is the updated SQLite file or if it is the same one as from FTP.

Answer 2

Apologies for the delayed response from over the weekend. Thanks for the follow-up and additional info.

Kindly follow the similar approach outlined in this GitHub discussion thread - The context for the discussion was when the database is in a VNET, but the approaches to extract data should help your situation as well. Essentially it boils down to using ssh and scp: [Featurerequest] Add section on how to connect to and query the database once deployedto Azure · Issue #34 · Azure-Samples/msdocs-flask-postgresql-sample-app(github.com).
As you mentioned above (/discussed), for production purposes kindly leverage hosted database (PostgreSQL, MySQL).

Share via

Getting Django SQLite file out of Web App

2 answers

Your answer