GSoC: Cloud Integration Preparation Work

May 12, 2016

After discussing API design, I’ve done some research on Dropbox, Google Drive and OneDrive API and discussed further steps with Peter and Eugene. I’ve identified some preparation steps I should do before GSoC starts. Doing this preparation work would also get me familiar with ScummVM code, coding & formatting conventions and commit guidelines.

I’m working on it in my public fork of scummvm repo on Github right in the master branch. Eugene helped me with configure (which I’m not using yet, as I’ve started with MSVC here) and watches how I’m doing.

My preparation work plan includes the following tasks:

We’ve decided that SimpleJSON is a good library to use as JSON parser in ScummVM. It’s really quite simple library, which consists of two classes and uses C++ standard library. It is now available as Common::JSON in my fork and uses ScummVM classes.

A few weeks back, I still was thinking that all cloud-related work would be done within separate thread spawned in main(). But when I’ve decided to actually add such thread (so my JSON examples could work and not delay ScummVM’s launcher), I understood that the original idea to use libcurl’s blocking functions is a bad idea. libcurl’s functions block thread execution until the request is complete. Because of that, our cloud thread won’t react to the user’s commands when it’s «busy». And that means if you started downloading 2 GB on 56 kbps, you can’t cancel it without killing ScummVM process!

That’s totally not what we want. There is no direct thread spawning in ScummVM and even though TimerManager actually runs callbacks in separate threads, such threads might work even when ScummVM main thread is finished (and TimerManager could be implemented so it won’t be running callbacks in separate threads anymore). So, I had to think of something. At first my idea was to add a callback through TimerManager, which would be executed every second. It would’ve checked whether the user wanted ScummVM to make some API requests and whether these requests were complete. But most of the time we don’t work with cloud, and thus such callback would be called even though it was unnecessary.

Now the idea is a bit more complex. We would have to use async libcurl functions, because we don’t want our callback to block. So, we can’t make our API methods to return the requested information, but instead those should receive some function pointer and call the passed function when this information is ready. I also believe that there could be more than one request pending, and some methods would require more than one REST API request.

Thus, I think of making special Request class, which would act as a base class for all methods implementations. For example, ListDirectoryRequest would be not only requesting the first «page» of directory contents, but make such requests until whole directory is listed. When all the information is gathered, it would call the callback.

Storage class, which would actually represent cloud storage API in ScummVM, has all the API methods we need. But these methods would be creating new Request objects only. The implementation would be hidden within these Request classes. So, Storage class would also contain some Request list. When it’s empty, there is nothing to do for Storage, and it does nothing. But when it is not, Storage would start a timer and poll these requests, whether they have complete what they were doing. When the request is complete, it is removed from this list. When list gets empty, Storage stops the timer.

Finally, there is also a CloudManager. This is a class, which would be loading all connected storages from ScummVM configs and providing fast access to the currently active Storage. So, the idea is to have all the storages loaded and ready, so user can easily switch between them in the Options menu. But only one Storage would be «active». This storage would be used to sync saves, upload and download files until user changes that. CloudManager would not only have a getter for it, but also a few «shortcut» methods, which would be using current storage within. For example, you won’t have to do cloudManager->getCurrentStorage()->syncSaves(). Instead, you can just do cloudManager->syncSaves().

I have not implemented Requests yet, but there are already some simple CloudManager and Storage stubs.

UPD: I’ve implemented first simple Request today. It’s not doing anything useful, but it shows that requests can «work» for a few handler() calls and then stop. When no requests are working, timer is automatically stopped.

Why do we need timer in the first place? Well, that’s because we want to know whether request is complete and we have to poll curl_multi_info_read to do that. Plus, this allows us to react to user’s commands. For example, user might want to cancel the download, and then Request would be stopped on the next timer tick.