The big Freeciv-web feedback and suggestions thread

Web version of freeciv at play.freeciv.org
louis94
Hardened
Posts: 227
Joined: Thu Apr 25, 2013 10:17 pm
Location: Belgium

Re: The big Freeciv-web feedback and suggestions thread

Postby louis94 » Mon Mar 02, 2015 9:38 pm

AndreasR wrote:So how can we improve the savegame system for Freeciv-web? I assume that continuing to use HTML5 Localstorage for the savegames still is a good idea, compared to the alternatives:
(…)
2. storing the savegames on the user's harddrive directly, which would not work on all devices (for example iPhones don't have a file-system where you can store savegames).
(…)
Let me know if you have any further suggestions!

I see one further alternative: allowing users to save their games to some cloud service (and keep a list of these so they may be loaded afterwards). OwnCloud may be the solution: it's open-source, WebDAV-based (thus easy JS integration) and everyone can run his own server.

uruiamme
Posts: 29
Joined: Sun Mar 01, 2015 6:05 am

Re: The big Freeciv-web feedback and suggestions thread

Postby uruiamme » Tue Mar 03, 2015 9:44 am

AndreasR wrote:I agree that the savegame system for Freeciv-web needs to be improved. [snip]

At the moment, the only way to free up space for savegames is to delete all the savegames. This can be done either when loading or saving a game. Unfortunately, there is no way to delete just some savegames at the moment.

[snip]

So please give me some time to make these improvements to the Freeciv-web savegame system, and then I hope you will have the possibility to test it once I have it released live.

I guess I can't take "no" for an answer!

I have been loading up some browser plug-ins for Firefox to see what I can come up with on deleting individual savegames. Well, so far I was able to see them using FireBug along with a plugin called FireStorage Plus! So far, the delete command I tried using FireBug did not work. The "deleted" savegame returned and I was even able to load the "deleted" game. I was going to try some of these recommendations... look at this website: http://stackoverflow.com/questions/3125980/firefox-addon-to-view-edit-create-localstorage-data (see Brock Adams reply from 2011) which leads me eventually to this webpage that talks about how to (hopefully) delete a single game... https://developer.mozilla.org/en-US/docs/Web/API/Storage/removeItem (This seems to indicate that we can maybe do this... I am wondering who is brave enough to try on a sample browser.) I also found info about SimpleStorage: http://libraries.io/bower/simpleStorage

Hack attempt, until the developer fixes the savegames...
  • Navigate to the domain in question, https://play.freeciv.org/
  • Open Firefox's console (Ctrl Shift K in windows).
  • Type localStorage.length This will make sure you have something in local storage
  • simpleStorage.index() This will make sure you have something in simple storage
  • simpleStorage.storageSize() will show you size
  • simpleStorage.get('username') This should output your username
  • simpleStorage.index() will show you a list of items stored... these are just the keys that can be queried or changed
  • simpleStorage.get('savegame-count') Outputs number of saves
  • simpleStorage.get('savegame-savename-6') This is the name of your 6th savegame
  • simpleStorage.get('savegame-file-6') This will be a lot of compressed data to fill the console.
  • simpleStorage.get('savegame-username-6') This will be your username.
  • simpleStorage.deleteKey('savegame-file-6') Deletes the 6th savegame, hopefully frees up space
  • simpleStorage.deleteKey('savegame-savename-6') Deletes the savegame listing title? Maybe we should rename it to say "DELETED" as a test??
  • simpleStorage.deleteKey('savegame-username-6') Deletes the rest of savegame 6.

The number of saves will now be messed up, and errors may occur due to the defunct savegame. My guess is it may be better to wipe these with NULL data, again, this is an untested hack... Here, instead of deleting the savegames, we can try to just reduce the size of the unwanted savegames. Of course, you may have 100 of them and need to do this a lot of times!!
  • simpleStorage.set('savegame-file-6', '') I have no idea if this would work, but the idea is to leave the key and set it to be blank. This is the key that stores the data.
  • simpleStorage.set('savegame-savename-6', 'DELETED') Changes the savegame #6 listing title? This is a test.
  • simpleStorage.storageSize() will show you size. Hopefully there is less than before!!!!

Also, another request or 2:
17.The UPGRADE command has no interface box to ask me to confirm an upgrade. If I have enough money, the upgrade is immediate and without being able to undo it.
18. I noticed in the Android version, the Settler always had a light-colored box around him that was the size and shape of a city's workable land area. This would be nice to see immediately which tiles a city will receive.
19. One of the major differences I noticed between these two games was that the number of food units can be listed like this: Suppose a city needs 12 food units to remain static, and there is a total of 14 units of food being produced. The Food field would say "14 (+2)" rather than just "14." I liked that a lot.
20. The happy/content people icons are extremely small. Are they able to be bigger?
Last edited by uruiamme on Wed Mar 04, 2015 1:37 am, edited 1 time in total.

louis94
Hardened
Posts: 227
Joined: Thu Apr 25, 2013 10:17 pm
Location: Belgium

Re: The big Freeciv-web feedback and suggestions thread

Postby louis94 » Tue Mar 03, 2015 4:44 pm

uruiamme wrote:The number of saves will now be messed up[/b], and errors may occur due to the defunct savegame.

From what I read in the code, deleted savegames will be still displayed as "undefined (undefined)". Trying to load them won't work, obviously (the client gets stuck).
In order to delete specific savegames, paste the following code in your browser's console:

Code: Select all

function listSavegames() {
    var total = simpleStorage.get("savegame-count");
    for (var i = 1; i <= total; ++i) {
        var name = simpleStorage.get("savegame-savename-" + i);
        var username = simpleStorage.get("savegame-username-" + i);
        console.log(i + ") " + name + " [" + username + "]");   
    }
}
function deleteSavegame(n) {
    var total = simpleStorage.get("savegame-count");
    for (var i = n + 1; i <= total; ++i) {
        var data = simpleStorage.get("savegame-file-" + i);
        var name = simpleStorage.get("savegame-savename-" + i);
        var username = simpleStorage.get("savegame-username-" + i);
        simpleStorage.set("savegame-file-" + (i - 1), data);
        simpleStorage.set("savegame-savename-" + (i - 1), name);
        simpleStorage.set("savegame-username-" + (i - 1), username);
    }
    simpleStorage.deleteKey("savegame-file-" + total);
    simpleStorage.deleteKey("savegame-savename-" + total);
    simpleStorage.deleteKey("savegame-username-" + total);
    simpleStorage.set("savegame-count", total - 1);
}

To get a list of the savegames (and their number), type:

Code: Select all

listSavegames()

To delete a specific savegame, type (N is the number of the savegame to remove)

Code: Select all

deleteSavegame(N)


Louis

uruiamme
Posts: 29
Joined: Sun Mar 01, 2015 6:05 am

Re: The big Freeciv-web feedback and suggestions thread

Postby uruiamme » Wed Mar 04, 2015 10:04 am

Thanks for the code. It appears that this was working. But it used up a lot of CPU time and memory... on the order of 15% CPU (maybe 1 processor) and 1.3 GB of memory. It also seemed to take over a minute to run for each deleted savegame.

But unfortunately, I am not sure what happened, but maybe I browsed to a different page at the wrong time. I noticed I was still on the End Game and options screen, so I navigated to the main Free Civ Web webpage. I am wondering if I foolishly did that while one of the games was deleting. Anyway, my storage is now 0 bytes and there is no Load Game option now. I guess I will restart with hopes that I manage the savegames better.

I noticed the script would be faster for the last of many savegames, but slower if you deleted game #1. It looks to me that moving all those games down 1 slot is pretty much a good way to crash something. It's amazing that something that is logically so simple can freeze up a modern computer doing such a simple task... it's like a massive register shift of a database that's just 5 MB. I think maybe the implementation is possibly writing and rewriting the file thousands of times 1 word at a time instead of storing things in logical chunks. Looks like this is such a rudimentary database that reads and writes are fine, but arbitrary changes to an entry take logarithmic time. Needs to be using a SQL database if you ask me.

uruiamme
Posts: 29
Joined: Sun Mar 01, 2015 6:05 am

Re: The big Freeciv-web feedback and suggestions thread

Postby uruiamme » Wed Mar 04, 2015 12:10 pm

Ahh, I restarted Firefox and the storage (and games) is back. I also did some time trials. With about 90 saves, it takes a minute to delete #1 and uses 1800 MB of memory. To delete #70, it's about 10 seconds and 400 MB. In both cases, it uses 17 to 20% of CPU time on my i7 processor, 4 cores/8 logical processors. And Firefox usually says and acts "not responding" after awhile. The script sometimes throws an error that says it is taking a long time. Restarting Firefox relatively clean improves things a little. Still, it would probably take an hour to remove one game at a time (always removing #1).

louis94
Hardened
Posts: 227
Joined: Thu Apr 25, 2013 10:17 pm
Location: Belgium

Re: The big Freeciv-web feedback and suggestions thread

Postby louis94 » Wed Mar 04, 2015 8:18 pm

This was intended more as a workaround than as a fix. I don't know why FF uses that much memory; maybe declaring the variables outside of the loop would help.
Maybe one of your savegames is corrupted (as the database syncing may have been interrupted).
A much more elegant solution would be not to display missing savegames. This would put a hard limit on the number of saves that can be done in one's life, but it would rarely be hit :D (On my computer, a C program would take 4 months just to count up to that number, 10^16)

uruiamme
Posts: 29
Joined: Sun Mar 01, 2015 6:05 am

Re: The big Freeciv-web feedback and suggestions thread

Postby uruiamme » Wed Mar 04, 2015 10:07 pm

Can you rewrite it, albeit less elegantly, with no string variables? In other words, can the math be done all inside the field where you have the "data" variable here?

Code: Select all

 simpleStorage.set("savegame-file-" + (i - 1), data);

Surely that will optimize the code if anything will. Thanks a bunch! I am doing some time trials. It's about 21 seconds to delete game #40 with about 80 savegames.

louis94
Hardened
Posts: 227
Joined: Thu Apr 25, 2013 10:17 pm
Location: Belgium

Re: The big Freeciv-web feedback and suggestions thread

Postby louis94 » Wed Mar 04, 2015 10:43 pm

uruiamme wrote:Can you rewrite it, albeit less elegantly, with no string variables? In other words, can the math be done all inside the field where you have the "data" variable here?

Sure! It's just a matter of moving a few letters around:

Code: Select all

function deleteSavegame(n) {
    var total = simpleStorage.get("savegame-count");
    for (var i = n + 1; i <= total; ++i) {
        simpleStorage.set("savegame-file-" + (i - 1), simpleStorage.get("savegame-file-" + i));
        simpleStorage.set("savegame-savename-" + (i - 1), simpleStorage.get("savegame-savename-" + i));
        simpleStorage.set("savegame-username-" + (i - 1), simpleStorage.get("savegame-username-" + i));
    }
    simpleStorage.deleteKey("savegame-file-" + total);
    simpleStorage.deleteKey("savegame-savename-" + total);
    simpleStorage.deleteKey("savegame-username-" + total);
    simpleStorage.set("savegame-count", total - 1);
}

The string variables will still be created as temporaries, so I doubt there will be any impact on performance.

User avatar
AndreasR
Elite
Posts: 702
Joined: Thu May 02, 2013 10:26 pm
Location: Norway

Re: The big Freeciv-web feedback and suggestions thread

Postby AndreasR » Wed Mar 04, 2015 10:55 pm

I have improved the savegame system for Freeciv-web, and plan to deploy a new version of Freeciv-web to the production server tomorrow if there are now big blockers.
The commit is here: Store savegames using XZ compression which results in smaller savegames. Allow uploading and downloading savegames to the local harddrive. Allow deleting oldest savegame. This new version will break compatibility with older savegames, but that is the price of progress. sorry in advance.

I also like the idea of OwnCloud a lot, but that is for a later time, maybe.

cazfi / louis94: do you know of any serious bugs in current Freeciv svn trunk preventing a deploy to production?

uruiamme
Posts: 29
Joined: Sun Mar 01, 2015 6:05 am

Re: The big Freeciv-web feedback and suggestions thread

Postby uruiamme » Thu Mar 05, 2015 7:32 am

So we can't load a previous game? Ahh. too bad. I was in the middle of a few.

Also, I did test the lastest in "deleting savegame hacks". LOL. Same issue with speed and memory use by Firefox. It appears to be disk bound... it's likely thrashing the I/O. And I noticed that Firefox doesn't release that memory very fast, so multiple deletes back to back can start to use a lot of memory.

As for the compression change, I guess you realize that the current file size is relatively small... 52 KB or so? I guess that gives you roughly 96 games at 5 MB. I thought bzip2 used to be one of the best compression routines.. or maybe 10 years ago it was. You know what I just thought of is that you may want to write a "savegame version" string someplace to allow the system to understand if there is an incompatible savegame stored in local storage.

Also, here is a bug report on the current game save. Please try to make this part of your testing:
  1. Suppose a player is in the middle of a lengthy game and has not saved periodically. The year is 1920.
  2. Suppose the Internet goes down, or the server crashes, or a battery dies, or someone cuts the fiber optic to the state of Arizona.
  3. The player will be presented with some obscure error message, which basically means "you lost everything you've done in this game recently"... see next couple of steps
  4. The player, assuming the best, tries to save the game.
  5. The game saves, and IT TELLS HIM THE GAME IS SAVED. YEA!
  6. The player will now restart the game, and load his saved game from the year 1920.
  7. However, the game state that is saved is a much earlier game state, essentially the same state as the game was started or last saved... like 1900.
  8. In other words, without the server connection, there appears to be no way to save a game. Can you remove this server dependency, which will also move the game toward being able to do autosaves?