GSoC: Project Summary
August 16, 2016
What I was working on during GSoC is Cloud storages support in ScummVM. Describing this feature in my proposal (mirror), I mentioned that it would include an API to interact with supported storage providers (which are Dropbox, OneDrive, Google Drive and Box), saves syncing mechanism, functionality to upload and download games data, and, of course, GUI for all of these. Proposal also has some extra tasks mostly making user experience better.
Some things were rediscussed during my work, but the main idea remained intact. The work is done and pull request already awaits final review before getting merged. API for all four announced storage providers works fine, saves are syncing and games data could be easily downloaded. Not only described extra tasks were complete, but also some functionality not mentioned in the proposal was added.
Saves sync is probably the main reason why Cloud storages support is needed at all. It allows users to easily continue playing the game on another device by simply connecting both to the same storage and doing the sync. It’s automatically started on ScummVM launch, on games saving (including autosaves) and when user opens Save/Load dialog. This dialog was updated to show a progress bar while syncing and also to «lock» slots which are being synced. To indicate that there is a sync in progress, small Cloud icon is shown in the corner.
To use the feature, users must connect a storage first. To do so, they should navigate into Cloud tab of Options dialog, select a storage provider and press «Connect» button. It opens a special Storage Connection Wizard, which provides the instructions on connecting. It has different variations depending on set of libraries ScummVM was compiled with. In the most simple case it says users should navigate to a special short link (to scummvm.org), which redirects them to provider’s page. When they allow ScummVM to use their storage on that page, they are redirected back to scummvm.org, where the code is shown. This code should be typed in the wizard dialog. It’s used by ScummVM to connect to the storage and use provider’s REST API then.
ScummVM page makes the code that way so wizard could check that code has no mistakes in it. If there is a mistake, it notifies user where it probably is. If ScummVM was built with SDL2, pasting from clipboard is supported. Wizard also has «Open URL» button, which makes it easier to navigate to provider’s page on platforms where URL opening was implemtented (these are Windows, Linux, Mac OS X, Android — iOS and Symbian are coming).
But it’s much easier to connect a storage when ScummVM’s built with SDL_net support, because then ScummVM runs a local webserver. In this case users are not redirected to ScummVM site from provider’s page — instead, they navigate directly to webserver’s page. No code typing is needed then, because ScummVM automatically gets it from user browser’s HTTP request. This webserver makes connecting a storage really fast and simple.
Another thing Cloud storage might be used for is games data download. Users can put their games into storage and then easily download on all their devices. A special «Download» button in Cloud tab opens Download Dialog, where users can select a remote directory to download and a local directory to download into. It shows a progress bar there and automatically tries to detect a game when download is complete. Users are also free to run download in background: no detection will happen, but a message will appear on the screen to notify them of finished download.
Both storage connecting and game downloading are shown in a video I’ve recorded. I’ve also posted information about my progress in the blog every week. Feature is documented on the wiki pages, with some diagrams included.
And, finally, we’ve decided that I should do a big extra task. Local webserver, which I originally proposed to simplify storage connecting process only, has been extended to be used for «Wi-Fi Sharing» feature. It means that while ScummVM’s local webserver is running, one can use browser on another device to navigate through directories, download files, create new directories or upload files!
Users can specify server’s port and their ’/root/’ directory within the same Cloud tab. Only files under specified directory and ScummVM’s saves directory are available, so users secure data is safe.
GSoC: Cloud security, WAGE updates
August 7, 2016
This week I’ve worked on some security-related updates for LocalWebserver. It now has a «minimal mode» — no Files Manager features are available, only redirect_uri. It’s used automatically in Storage Connection Wizard, and server goes offline the moment it gets the code.
Paths are now dealt with more strictly: «../» is forbidden to use, ScummVM has some «blacklisted» directories and user can define where the «/root/» is. Files outside that folder are not available through Files Manager. Plus, if no rootpath specified, «/root/» is not even listed — only «/saves/» is available then. GUI for changing rootpath is added to the same Cloud tab.
One other thing I did was rebasing. Travis checks PRs by merging with master and then building. Titanic, which wasn’t there when I started, was using OutSaveFile as if it’s typedefed WriteStream. But in my Cloud PR OutSaveFile is a real class (which starts saves sync when finalized). So, I had to rebase and add a simple fix in Titanic, and now Travis checks are passed again.
In order to update WAGE saves, I had to add pos() into WriteStream class. Managed to break a few builds that way.
Also, fixed a few crashes in WAGE games. One was because code tried to copy pixels outside of the Surface, and I added checks, so copied rectangle is clipped to always be within surface area. The other was because operator was used on an empty String, and now that code uses it only when String it not empty.
GSoC: Joined the Team
July 31, 2016
Well, this week I was mostly working on some fixes for Cloud PR, some minor warnings, TODOs. Updated GUI a little bit: added a «Paste» button in Storage Connection Dialog, so user’s clipboard could be automatically put in the 8 code fields. Dialog also contains a Container now, because buttons couldn’t fit in 320×200. Storage providers logos are now shown, which is just a nice thing to have.
Apart from that, I’m waiting for PR feedback, and thus have to do something while waiting. I’ve joined the ScummVM team and now working on WAGE engine. Well, the only thing I did for now was «porting» save/load code from existing Java implementation of WAGE engine into our ScummVM one.
GSoC: The PR
July 24, 2016
(I don’t have any excuse, I just forgot to post that in time.)
This week I’ve sent my Cloud storage support as pull request to the main ScummVM Github repository, and thus started fixing everything what is not fixed enough, adding new minor features, etc.
Only pasting from clipboard support was added for EditableWidgets (I’d have to fix that too, btw).
I’ve added two new test suites in the Testbed: Cloud and Webserver. I’ve also added one test for openUrl in MiscTests. Almost all my tests are interactive, meaning they require user to check whether the feature works as it should, and press the right button.
Peter Bozsó (uruk-hai), my mentor, added OS X backend for openUrl. iOS backend it not there yet.
Another small feature which lacks iOS backend is the Networking::Connection::isLimited() function, which is used in DownloadDialog to notify users to think whether they want to download game data while using limited connection. For now only Android supports this, other platforms are using default backend, which always returns that connection is not limited.
AJAX version of local webserver’s Files Manager was added this week. One can navigate through directories without page refreshing all the time. This version also contains «breadcrumbs» navigation feature, so one can get up more than one level with one click at the path.
Users can override webserver’s port through the Options dialog now. Yet, if they want to use webserver for auth, they have to use the default value.
Some refactoring/cleanup in the code and different minor fixes were made. More OSD messages added to notify user of success or failure. I’ve also updated Dropbox to use their API v2 everywhere, as v1 is deprecated now.
I’ve added detailed information about the cloud storage support and local webserver to the ScummVM wiki. These should help people understand PR’s code. Some ideas are coming out in the discussion. For example, webserver is being stopped now when users close the Options dialog to prevent it being used by someone else.
GSoC: GUI, Box and Two WIP Features
July 17, 2016
In my previous post I’ve mentioned my Container PR — and voila — it was merged! So the first thing I’ve done this week was merging it from upstream’s master into my cloud fork’s master and using the Container in the Cloud tab. I’ve also fixed a few TabWidget’s height issues in that a bit later.
Box support was added this week, so the original plan to add 4 cloud services is accomplished. Box is a bit similar to Google Drive, as it uses files ids instead of paths. The other interesting thing about it is that it has yet another approach to file uploading, so I had to tune some things one more time to support this one too.
I’m not planning to add more cloud services yet. I was unable to even find out how to work with iCloud. OwnCloud uses WebDAV instead of OAuth2 + REST API and they are not going to implement one. I’m not sure how long will it take to add WebDAV support, so I’d be working on other things for now.
A few upgrades were made in the DownloadDialog and local webserver’s Files Manager. The first one now shows progress a bit more precisely because it’s calculated based on downloaded size, not the number of files downloaded. It also shows the downloaded size, total size and current speed there.
Files Manager now supports directories uploading (works in Chrome only, as that’s the only browser where this feature is implemented). I’ve also added code which should correctly determine server’s IP on Linux. Some minor improvements like file type icons were made:
Finally, I started working on openUrl and clipboard support. Windows, Linux and Android already support opening URL in the browser, OS X and iOS backends are on the way. I also have some Symbian code, but it could take some time to implement it properly.
Clipboard is already supported in SDL2, and I just started working on adding it. Right now it’s only Ctrl+V support for EditableWidget. We’re also thinking on adding selecting feature there, so users could also use Ctrl+C. I’m not sure how that should work on platforms like Android, where we usually don’t have keyboards attached. I mean, EditableWidgets are not Android EditText fields, so they don’t show a menu on long tap, which means we either implement it or make users suffer.
These two features are what I’m going to work on. Then I also should add some tests into testbed engine, and do some documenting/refactoring where needed.