Open Source augmented reality project: PapARt 2.0


Augmented Reality made easy.

The goal of the PapARt project was to create easily Spatial Augmented Reality applications. Consequently the first versions are large and complex to use because it does a lot for an AR framework.

We propose to split the current PapARt project to different sub-projects powered by RealityTech and this community.


The main goal is to divide the software in many components that work together.

Camera handling.

  • One part of the software handles the camera stream. OpenCV, FFMPEG and many other libraries do an awesome job at that already in PapARt. Let’s add Android, and raspberry pi cameras. This is what we do thanks to JavaCPP-presets in Papart in the Camera package.

Feature detection.

  • Another part does the marker/features detection. This part requires a model with the sizes of the objects to finds and algorithms to find this object in the video stream. In PapARt we do it already with ARToolkitPlus and OpenCV natural features in this package. We would like to extend it with NyARtoolkit to have detection without native code, Reactivision fiducials, and ChiliTags. All of them have Java packages already. We also would like to integrate commercial tracking libraries.

Network communication.

  • These two parts are managed by a server that can stream the 3D positions found by the marker tracking. This server will also provide the camera intrinsic parameters for OpenGL / DirectX rendering.

Image / object analysis.

  • An image / depth analysis framework will work on top that. This could extract color information, shapes, 3D elements (when using a depth camera).

Multiple 3D engines.

  • As the tracking and the cameras are external programs, any program could connect to get the video feed and 3D locations. Consequently, we will create AR clients for Processing, and Unity3D. We will encourage all the communities to join and create their own client: vvvv, openFrameworks etc… The Unity3D package is likely to have a free version and a paid version with extra features. This will be discussed later in the “how to finance” threads.

We (RealityTech) want to build this as part of a larger framework for our internal use. We commit to create open source solutions as much as possible. AR tracking, detection, rendering and camera handling are problem that will stay for a long time. We propose to build a platform that builds up between analysis (OpenCV / PCL) and 3D engines (Processing, Unity3D).

So when you start your project, you could find the marker tracking that fits your needs, write the image analysis in the language your prefer, and use a 3D engine adapted to your needs.