[dev] How we got Travis Continuous Integration for multiple language projects in a mono repo

With every commit that makes it to the monorepo both the Android and Desktop project are built and tested. This post documents how we did that.

This post is intended to help developers on other projects where they need wish to start multiple travis jobs for completely different configurations.

Most documented examples on the web on how to use travis are usually focused only on building a single project in a single environment. But what if your repository has multiple projects that need to run under different contexts and programming languages.

At FrostWire we have a big monorepo which is host to multiple projects:

android/ # android client (java8, android-sdk environment gradle build)
desktop/ # desktop client and tests (java11+ gradle build)
common/ # common java sources for android and desktop projects


Travis allows you to define what we understand to be a “Job Matrix”, where each entry in this Matrix specifies an individual job (and they can run in parallel when you push your commit and travis gets word of it)

Here’s our first version of .travis.yml to build the android and desktop projects, and run tests jobs on a single git push:
matrix:
  include:
    # android build job
    - language: android
      dist: xenial
      android:
        components:
          - build-tools 29.0.2
          - android-29
          - extra-google-google_play_services
          - extra-google-m2repository
          - extra-android-m2repository
      licenses:
        - 'android-sdk-preview-license-52d11cd2'
        - 'android-sdk-license-.+'
        - 'google-gdk-license-.+'
      script:
        - yes | $ANDROID_HOME/tools/bin/sdkmanager "build-tools;29.0.2"
        - cd android
        - pwd
        - ./gradlew assembleDebug
    # desktop build job
    - language: java
      jdk:
        - openjdk11
      script:
        - cd desktop
        - gradle build
    # desktop/common tests job
    - language: java
      jdk:
        - openjdk11
      script:
        - cd desktop
        - gradle test

A New FrostWire 6.1.2 for PC, MacOSX and Linux is yours to download.

Download Installers on FrostWire’s Official Website
Download Installers and SourceCode on GitHub

Screenshot of Changelog in Github.com

Full Changelog:

frostwire (6.1.2) stable; urgency=high
  * Search performance improvements. New RxJava-based implementation.
  * Search box suggestions improved.
  * New Check/uncheck all search engines checkbox.
  * TorLock search fixed.
  * Added SSL (https) support for torrent web seeds.
  * Options window remembers last pane opened.
  * Fixes painting issues when searching Starred audio files on the library.
  * frostwire-jlibtorrent library updates.
  * Library user experience bug fixes.
        
 -- FrostWire Team   Wed, 24 Jun 2015 11:44:00 -0500

Write runtime libtorrent plugins with Java using frostwire-jlibtorrent #FrostWireLabs

We’ve just pushed upgrades to the frostwire-jlibtorrent API that will allow developers to extend and interact the Bittorrent protocol at a lower level through the implementation of libtorrent plugins.

Before today, a libtorrent plugin developer had to code plugins in C++ and rebuild the libtorrent library binaries for the architectures on which the plugin was meant to be used.

We think that non-bittorrent higher level abstractions (e.g. Bitcoin+Bittorrent integration) that need to occur near the core of the bittorrent protocol can be more easily coded, tested and debugged in a language like Java, without having to touch nor recompile the libtorrent libraries.

Today we’ve made available a new portion of the frostwire-jlibtorrent API that allows you to do this without all the C++ development headaches, you just focus on your plugin logic, and your code will also be available in all platforms with the same .class files.

Here’s an example of a dummy libtorrent Plugin implemented in Java using frostwire-jlibtorrent

To create a Plugin all you need to do is extend the AbstractPlugin class, whose newTorrent method will ask you to provide a TorrentPlugin interface implementation, this is meant to be a plugin instance that takes care of what happens when a torrent is added to the libtorrent session.

We provide an AbstractTorrentPlugin class for you to extend and as you do this you may or may not need to provide your own implementation of the TorrentPlugin::newPeerConnection method, which is meant to return a PeerPlugin, a plugin instance that takes care of what happens for every Peer connection, where probably most of your plugin logic will occur. We also provide an AbstractPeerPlugin implementation that you can extend.

Once you have your plugin ready to go, you can add it to the session with session.addExtension(myPlugin) and all torrents added to the session will invoke hooks that could be handled by your plugin implementations.

New FrostWire 5.7.5 for your desktop, now Java8 powered.

Download FrostWire 5.7.5 for Windows (Bitcoin, Litecoin, Dogecoin enabled .torrent)
Download FrostWire 5.7.5 for MacOSX (.torrent)
Download FrostWire 5.7.5 for Debian/Ubuntu (.torrent)

FrostWire 5.7.5 runs on Java 8, expect a faster and improved FrostWire overall.

frostwire-575-poly

We’ve upgraded our mp4 parsing libraries, this should solve some possible errors at the end of some media downloads.

We’ve also improved the quality of our cloud search results specially for music and video files.

Changelog

frostwire (5.7.5) stable; urgency=high

  * Improved mp4 parser.

  * Improved cloud search results.

  * Now FrostWire runs with Java8 technology.

  * Fixes player control layout issues when resizing the main window.

  * Fixes soundcloud url track/set downloads.

  * Fixes possible cloud download errors.

 

FrostWire is a free, open source BitTorrent client first released in September 2004, as a fork of LimeWire. It was initially very similar to LimeWire in appearance and functionality, but over time developers added more features, including BitTorrent support. In version 5, Gnutella support was dropped entirely, and FrostWire is only a BitTorrent client. Development of the program has been active since the program was first released in September 2004.

FrostWire for Desktop’s Lines of Code Count

    8724 text files.
    8237 unique files.                                          
    8917 files ignored.

http://cloc.sourceforge.net v 1.60  T=121.17 s (65.3 files/s, 14184.1 lines/s)
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
Java                           7591         300370         340020        1033777
HTML                            192           2811           2130          21190
C++                              12           1337            510           3207
XML                              17            178            101           1632
C/C++ Header                     20            562           1623           1188
Ant                              19            289            136           1124
Maven                            10             66             20           1073
JSP                              16             64            606            777
Objective C                       7            184            100            675
Objective C++                     1            138             71            583
Javascript                        5             58             25            361
SQL                               1             34              7            288
CSS                               2             63             13            274
Bourne Again Shell                3             45             25            229
XSD                               2             28             57            187
C                                 2             58             14            134
PHP                               1             18              0             73
Bourne Shell                      4             13              1             45
DOS Batch                         7              8              4             41
make                              1              9              0             21
--------------------------------------------------------------------------------
SUM:                           7913         306333         345463        1066879
--------------------------------------------------------------------------------

Help us over a million lines of code at https://github.com/frostwire

FrostWire 5.7.0 is out for Windows, Mac and Linux now at FrostWire.com

Download FrostWire 5.7.0 for Windows (torrent)

Download FrostWire 5.7.0 for Mac (torrent)

Download FrostWire 5.7.0 for Ubuntu (torrent)

Download FrostWire 5.7.0 for Red Hat/Cent OS/RPM

Download FrostWire 5.7.0 for any other Linux Distro

Salsa Sara hoops in Wynwood, Miami for the FrostWire Team
Salsa Sara hoops in Wynwood, Miami for the FrostWire Team

What’s new?

  • Updated BitTorrent Engine to the latest code from the Azureus project along with FrostWire’s Team’s tunning and hacks for energy efficiency (as we reuse this same on code on Android)
  • Updated Java Runtime to fix compatibility launch issues Windows XP users were experiencing.
  • IRC Chat replaced for an HTML5 Chat App (Kiwi IRC), bye bye pjirc client.
  • New Torrents.com search engine integration.
  • Several UI bugs, freezes and crashes fixed.

Stay tuned for our Android release…

How to help translate FrostWire for Desktop (Windows / Mac / Linux)

Help us translate the FrostWire user interface, be part of an open source project, learn new things.

0. Get a GitHub.com account and sign in
Go to GitHub.com and sign up for a free account unless you have one already.

1. Download and install Git on your computer
If you don’t have git installed on your computer, here are instructions on how to install.

git, is a version control software which helps us keep track of all the changes on every file that belong to the FrostWire project.
GitHub.com is a site where we host our source code and its origin git repository (the official one), think of the repository as a database to keep all those file versions.

GitHub is really helpful because it makes colaboration very social, we can comment and review each other’s changes before merging them into the origin repository.

2. Fork us on github.
Go to https://github.com/frostwire/frostwire-desktop/, this is the page for our origin repo. Forking means you will be making a copy of your own inside your github account. You will wok on that one, and when you are done you will send the changes to the origin repo.

For us by clicking on the button that says “Fork” on the top right.

Once the fork is done, you can go to your github personal page, and in the list of your repositories you should have your frostwire-desktop fork.

As of now, that copy lives only at github.com, you could try and edit the files up there, but it’s very uncomfortable working that way, the text editor can be quite slow sometimes, so …

3. Clone your forked repo to your computer
Go to a command line, or with your favorite git client make a clone of YOUR repo (not ours).
If you use the command line git client, you should issue the following command

git clone https://github.com/myusername/frostwire-desktop

wait a few seconds and all the source code and assets that belong to the frostwire-desktop project will be downloaded to a folder called “frostwire-desktop” on your computer.

change directory to it…
cd frostwire-desktop

4. Create a branch for your translation
Create a branch with a name that will help us understand that this is a translation update you’re sending us, it could be named something like
“translation-french-2013-november-myname”, so that we can easily see what language you are translating, on which date you were working on it, and your name so we can give you credit for it.

You create a branch on the git command line like this (just remember to put the name of YOUR branch instead)

Create a branch locally
git branch translation-french-2013-november-myname

Push it to your remote repository
git push origin -u translation-french-2013-november-myname

now your branch lives both on your computer and at your github repository.

Switch to that branch
git checkout translation-french-2013-november-myname

Now you’re ready to start working on your branch.

After we’ve accepted your changes, next time you want to help us do another update of the translation you will have to create a new branch.

5. Make sure the latest english strings have been put into your language.po file
enter the following command to sync strings

ant gettext-extract

(this could take a while to finish as it goes through every line of code looking for translatable strings.)

6. Translate, translate, translate
Now you get to do the actual translation. Translation files are in the following folder inside the frostwire-desktop project
cd lib/messagebundles

If you’re going to translate say to arabic, you will edit the ar.po file, it should be easy to find the language file for the language you intend to work on. We recommend that you use a PO Editor software to make things easier, and always make sure to save the file using UTF-8 encoding.

7. Test your translation
To test your translation, you will need to recreate the message bundle file, for this you will need to invoke the following command

ant gettext-bundle

assuming you didn’t mess up anything in the format of the .po file this should finish after a few minutes of bundling every language file.
If you see any errors you should try to fix them, if you don’t know how to fix them you can reach us on the FrostWire forum, or right here for help.

once the bundle is built, you need to test your translation, for this you will need to build FrostWire, you can do this in one step, from the root frostwire-desktop folder type:

ant

after a couple minutes it should be done compiling everything, and then you can invoke the “run” script right there if you’re on Linux or Mac, if you’re on windows, go to gui/ and invoke the run.bat file.

FrostWire will open, switch to your language and make sure your translations are fine.

8. Time to commit and push your changes

Once you are finished, commit your changes, make sure you don’t make changes in other files than the .po of the language you are working with.
So if you’re working say with italian, you would do (from the frostwire-desktop root directory)

git commit lib/messagebundles/it.po -m "my translation update for italian users"

and then push it to your fork up on github

git push

9. Submit a pull request

Once you see your last commit on github and you are sure you’re finished, it’s time to let us know, so you will submit what’s called a “Pull Request”.
If there are any special notes please let us know, we’ll review your changes, and if everything is good to go, we’ll merge them, and you will make open source history 🙂

Configuring your Eclipse Build Path to build FrostWire

Everytime I setup eclipse in Windows I end up wasting about 30 minutes configuring the build path to get rid of all errors, here’s a self-reference post that you can also use to configure your Eclipse and get it ready to hack and patch FrostWire.

Source Tab
Add frostwire/core folder
Exclude: examples/**
Native library location: frostwire/lib/native/windows

Add frostwire/gui folder
Exclude: com/limegroup/gnutella/gui/Mac*.java; macosx; plugins; windows;
Native library location: frostwire/lib/native/windows

Add frostwire/gui/windows folder (nested, yes)

Libraries Tab
Add all jars in lib/jars, frostwire/dist, frostwire/components/all/dist

Order and Export Tab
Elements should be ordered as follows:

1. frostwire/core folder
2. frostwire/gui folder
3. frostwire/gui/windows folder
4. JRE System
5. lw-all.jar
6. FrostWire.jar
… Rest of jars.