claritylab / lucida

Lucida Lucida is a speech and vision based intelligent personal assistant inspired by Sirius. Visit our website for tutorial, and Lucida-users for help. The project is released under BSD license, except certain submodules contain their own specific licensing information. We would love to have your help on improving Lucida, and see CONTRIBUTING for more details. Overview • : back-end services and command center (CMD). Currently, there are 7 categories of back-end services: "ASR" (automatic speech recognition), "IMM" (image matching), "QA" (question answering), "CA" (calendar events retrieval), "IMC" (image classification), "FACE" (facial recognition), and "DIG" (digit recognition). You can delete or replace these services with your own, or you can simply add a new service. For example, if you know some better ASR implementation, have an interesting image captioning end-to-end system, or have access to a quality machine translation algorithm, please read the section "How to Add Your Own Service into Lucida?" below. The command center determines which services are needed based on the user input, sends requests to them, and returns response to the user. In the following diagram, the user asks a query that needs the following three services: ASR, IMM, and QA. The "cloud" behind each box means the Docker container(s) running on the host machine(s). • : dependencies necessary for compiling Lucida. Due to the fact that services share some common dependencies, all services should be compiled after these dependencies are installed. The advantage of a central point of dependencies is that the total size of compiled services is minimized; the disadvantage is that it makes deleting a service from Lucida non-trivial -- you have to remove its dependencies in . Lucida Local Development If you want to make contributions to Lucida, please build it locally: • From this directory, type: . This will run scripts in to install all the required dependencies. After that, it will compile back-end services in . • Important note for Ubuntu 16.04 users: please read note #1. • If for some reason you need to compile part of it (e.g. one back-end service), make sure to set the following environment variable as set in : You can add it permanently to your bash profile. • Start all services: This will spawn a terminal window ( ) for each service as well as the command center. Once they all start running, open your browser and visit . Check out the for usage and sample questions. Currently, the command center receives the user input in the form of HTTP requests sent from your browser, but in future we can support other forms of input. Lucida Docker Deployment If you want to use Lucida as a web application, please deploy using Docker and Kubernetes: • Install Docker: refer to https://docs.docker.com/engine/installation/. • Navigate to and follow the instructions there. • Once done, check out the for usage and sample questions. REST API for command center The REST API is in active development and may change drastically. It currently supports only infer and learn. Other features may be added later. An example client for botframework is available. Information on how to use the API can be found in the wiki Design Notes -- How to Add Your Own Service into Lucida? Back-end Communication Thrift is an RPC framework with the advantages of being efficient and language-neutral. It was originally developed by Facebook and now developed by both the open-source community (Apache Thrift) and Facebook. We use both Apache Thrift and Facebook Thrift because Facebook Thrift has a fully asynchronous C++ server but does not support Java very well. Also, Apache Thrift seems to be more popular. Therefore, we recommend using Apache Thrift for services written in Python and Java, and Facebook Thrift for services written in C++. However, you can choose either one for your own service as long as you follow the steps below. One disadvantage about Thrift is that the interface has to be pre-defined and implemented by each service. If the interface changes, all services have to re-implement the interface. We try to avoid changing the interface by careful design, but if you really need to adapt the interface for your need, feel free to modify, but make sure that all services implement and use the new interface. Detailed Instructions You need to configure the command center (CMD) besides implementing the Thrift interface in order to add your own service into Lucida. Let's break it down into two steps: • Implement the Thrift interface jointly defined in and . • The basic functionalities that your service needs to provide are called , , and . They all take in the same type of parameters, a representing the Lucida user ID ( ), and a defined in . The command center invokes these three procedures implemented by your service, and services can also invoke these procedures on each other to achieve communication. Thus the typical data flow looks like this: But it also can be like this: In this scenario, make sure to implement the asynchronous Thrift interface. If YOS0 implements the asynchronous Thrift interface, it won't block on waiting for the response from YOS1. If YOS0 implements the synchronous Thrift interface, it cannot make progress until YOS1 returns the response, so the operating system will perform a thread context switch, and let the current thread sleep until YOS1 returns. See section 3 of step 1 for implementation details. : create an intelligent instance based on supplied LUCID. It gives services a chance to warm up the pipeline, but our current services do not need that. Therefore, the command center does not send request at this point. If your service needs to warm up for each user, make sure to modify the command center which is detailed in step 2. : tell the intelligent instance to learn new knowledge based on data supplied in the query, which usually means the training pro…