Skip to content

Modifications to SqliteStorage to allow for custom filepaths #77

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ErikBjare merged 2 commits into from
May 21, 2019
Merged

Modifications to SqliteStorage to allow for custom filepaths #77

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d0714e2
modifications to SqliteStorage to allow for custom filepaths (outside…
ErikBjare May 6, 2019
bacf2d7
fixed accidental constant change
ErikBjare May 6, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions aw_datastore/migration.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@

logger = logging.getLogger(__name__)


def detect_db_files(data_dir: str, datastore_name: str = None, version=None) -> List[str]:
db_files = [filename for filename in os.listdir(data_dir)]
if datastore_name:
Expand All @@ -16,17 +17,19 @@ def detect_db_files(data_dir: str, datastore_name: str = None, version=None) ->
db_files = [filename for filename in db_files if filename.split(".")[1] == "v{}".format(version)]
return db_files

def check_for_migration(datastore: AbstractStorage, datastore_name: str, version: int):

def check_for_migration(datastore: AbstractStorage):
data_dir = get_data_dir("aw-server")

if datastore.sid == "sqlite":
peewee_type = "peewee-sqlite"
peewee_name = peewee_type + "-testing" if datastore.testing else ""
peewee_name = peewee_type + ("-testing" if datastore.testing else "")
# Migrate from peewee v2
peewee_db_v2 = detect_db_files(data_dir, peewee_name, 2)
if len(peewee_db_v2) > 0:
peewee_v2_to_sqlite_v1(datastore)


def peewee_v2_to_sqlite_v1(datastore):
logger.info("Migrating database from peewee v2 to sqlite v1")
from .storages import PeeweeStorage
Expand Down
29 changes: 19 additions & 10 deletions aw_datastore/storages/sqlite.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,13 +57,19 @@
class SqliteStorage(AbstractStorage):
sid = "sqlite"

def __init__(self, testing):
def __init__(self, testing, filepath: str = None, enable_lazy_commit=True) -> None:
self.testing = testing
data_dir = get_data_dir("aw-server")
self.enable_lazy_commit = enable_lazy_commit

# Ignore the migration check if custom filepath is set
ignore_migration_check = filepath is not None

ds_name = self.sid + ('-testing' if testing else '')
filename = ds_name + ".v{}".format(LATEST_VERSION) + '.db'
filepath = os.path.join(data_dir, filename)
if not filepath:
data_dir = get_data_dir("aw-server")
filename = ds_name + ".v{}".format(LATEST_VERSION) + '.db'
filepath = os.path.join(data_dir, filename)

new_db_file = not os.path.exists(filepath)
self.conn = sqlite3.connect(filepath)
logger.info("Using database file: {}".format(filepath))
Expand All @@ -77,10 +83,10 @@ def __init__(self, testing):
self.conn.execute("PRAGMA journal_mode=WAL;")
self.commit()

if new_db_file:
if new_db_file and not ignore_migration_check:
logger.info("Created new SQlite db file")
from aw_datastore import check_for_migration
check_for_migration(self, ds_name, LATEST_VERSION)
check_for_migration(self)

self.last_commit = datetime.now()
self.num_uncommited_statements = 0
Expand All @@ -102,10 +108,13 @@ def conditional_commit(self, num_statements):
This is because sqlite is very slow with small inserts, this
is a way to batch them together and lower CPU+disk usage
"""
self.num_uncommited_statements += num_statements
if self.num_uncommited_statements > 50:
self.commit()
if (self.last_commit - datetime.now()) > timedelta(seconds=10):
if self.enable_lazy_commit:
self.num_uncommited_statements += num_statements
if self.num_uncommited_statements > 10:

@johan-bjareholt johan-bjareholt May 6, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why lower it from 50 to 10?

And why do you need the enable_lazy_commit at all? It should work transparently?

@ErikBjare ErikBjare May 6, 2019 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, the 50 to 10 was accidental. Fixed.

enable_lazy_commit=True effectively makes the code asynchronous, which wasn't acceptable for a use-case I had (sync).

@johan-bjareholt johan-bjareholt May 6, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understood that the use-case was sync, but why do you really need it?

Why do you need the absolute latest event? If you actually do, can't you instead force a commit when you actually need it like when using the REST API?

@ErikBjare ErikBjare May 6, 2019 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you need the absolute latest event?

Because I'm checking for the latest sent event sometimes, to know which events to send (I'd need to know about eventual uncommitted events).

If you actually do, can't you instead force a commit when you actually need it like when using the REST API?

I am calling it through the ServerAPI class in aw_server/api.py, and there doesn't seem to be any commit done through the REST API? How would you force a commit through the REST API?

@johan-bjareholt johan-bjareholt May 6, 2019 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/ActivityWatch/aw-server/blob/master/aw_server/api.py#L53
https://github.com/ActivityWatch/aw-core/blob/master/aw_datastore/storages/sqlite.py#L224

If that's the case you should already be getting the latest event

@ErikBjare ErikBjare May 6, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, that's weird, because I definitely didn't (That's why I did this in the first place). I'll take a closer look at it.

@johan-bjareholt johan-bjareholt May 6, 2019 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it's because you are running it as a seperate process?

Same thing here as with aw-server-rust, you should not be accessing a SQLite database from multiple threads or processes. This can corrupt the database if not configured properly.

@ErikBjare ErikBjare May 6, 2019 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it's because you are running it as a seperate process?

I'm not, I'm instantiating two ServerAPI objects with two different SqliteDatastore objects (with different db paths) in the same process and thread.

I've reproduced the issue by setting enable_lazy_commit=True, not sure what is going wrong, only seems to happen if process dies shortly after commit?

@ErikBjare ErikBjare May 21, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't figure it out, I'm going ahead and merging this anyway.

self.commit()
if (self.last_commit - datetime.now()) > timedelta(seconds=10):
self.commit()
else:
self.commit()

def buckets(self):
Expand Down