Thanks to researchers from DeepMind, you can now build your own transformer models from scratch! A college student created an app to recognize text written by AI and an AI bot beat the Rocket League elite. Vision-based ML Is helping pilots land planes and we’re sharing how you can use Lightning to train a GPT model with 20-billion parameters for text prediction.
Research Highlights:
To speed up the interpretability research, DeepMind researchers introduced Tracr. Tracr is a “compiler” that converts RASP, a domain-specific language, code into weights for a standard, decoder-only, transformer architecture that is similar to the GPT. It takes RASP code written in a human-readable format and converts it into weights for a transformer model.
Researchers from MIT and University of Texas released a preprint of their research on on the capabilities of large language models (LLMs) from perspective of cognitive science. On the one hand, the authors contend that LLMs are surprisingly adept at formal linguistic competence, making major advancements in their understanding of phonology, morphosyntax, etc. However, LLMs still struggle with most functional skills (math, reasoning, and general knowledge), especially when they break from textual conventions. The word-in-context prediction objective is thus claimed to be insufficient for mastering human mind, yet being surprisingly effective at learning a great deal about language.
GLIGEN, or Grounded-Language-to-Image Generation, is a method developed by Microsoft researchers that expands and builds upon the capabilities of pre-trained text-to-image diffusion models by allowing them to also be conditioned on grounding inputs. With caption and bounding box condition inputs, their model is said to generate open-world grounded text2img, and the grounding ability generalizes well to novel spatial configurations and concepts. According to the authors, GLIGEN’s zero-shot performance on COCO and LVIS outperforms the supervised layout-to-image baselines by a significant margin.
ML Engineering Highlights
GPTZero, a web app built by a Princeton student, claims it can determine if an essay was produced manually or by ChatGPT, an artificial intelligence chatbot. This might be used to spot informational misrepresentation and academic dishonesty, but only if OpenAI, the firm behind the well-known chatbot, keeps granting access to the underlying AI algorithms. Since GPTZero was trained using datasets comparable to those used by ChatGPT, it is able to assess the relative complexity, or “perplexity” reading, of a text. A text is more likely AI-generated if it is familiar to the GPTZero bot.
The testing of new pilot support systems with an A350-1000 test aircraft has begun, according to a recent announcement from Airbus UpNext. The three technologies being tested are automatic landing, automated emergency diversion in cruise, and taxi assistance. Airbus UpNext also announced that it is launching a project to advance next-generation computer vision-based algorithms in support of landing and taxi assistance.
An AI-driven retail technology created by Google Cloud enables merchants to use imagery to automatically track items on store shelves. The system is now in preview and is expected to be broadly accessible “in the coming months,” according to the internet giant. It uses a Google library with billions of data points to identify objects in images shot from various angles.
Open Source Highlights
Cheaters Beat the Rocket League elite by hacking an AI Bot. Rocket League is notoriously tricky to play and requires considerable skill and patience. The bot that is beating the best players picked up the ability to dribble and score using reinforcement learning. The company behind Rocket League, Psyonix, part of Epic Games, allows players to deploy bots to practice against. In 2020 it made an API available to help developers build bots more easily. Last April, a group of Rocket League enthusiasts with coding skills announced RLGym, an open source library for building reinforcement-learning bots for Rocket League.
Tutorial of the Week
This week, we shared how you can use Lightning to train a GPT model with 20-billion parameters for text prediction. You can train your model on the cloud and customize the hardware to your use case with just a single flag.
Community Spotlight
Want your work featured? Contact us on Discord or email us at [email protected]
- This recently-merged community PR addresses a bug related to multiple inheritance and mixins. You can also read about this on the GitHub issue here.
- This PR fixes an issue related to forked subprocesses. The related GitHub issue is also a stellar example of how we love to see open-source work being done: collaboratively and efficiently!
- Finally, this PR moves both pytest and coverage configuration from the setup.cfg file to pyproject.toml in order to minimize overall project structure.
Kudos to another week of amazing work by our community!
Lightning AI Highlights
The Lightning 1.9 release is out! This release includes a number of stability improvements and bug fixes and adds additional support for training strategies like FSDP and DDP.
Forums are back! In addition to the public Lightning Discord workspace, pop over to our dedicated forums to ask questions and respond to other users. Here’s to building together in the future!
Don’t Miss the Submission Deadline
- ACL 2023: The 61st Annual Meeting of the Association for Computational Linguistics. July 9-14 2023 (Toronto, Canada). Full paper submission deadline: January 20, 2023
- ICML 2023: Fortieth International Conference on Machine Learning. Jul 23-29 (Honolulu, Hawaii). Full paper submission deadline: January 26, 2023 08:00 PM UTC
- IROS 2023 :International Conference on Intelligent Robots and Systems. Oct 1 – 5, 2023 (Detroit, Michigan). Full paper submission deadline: March 1, 2023
- ICCV 2023: International Conference on Computer Vision. Oct 2 – 6, 2023. (Paris, France). 1. Full paper submission deadline: March 8, 2023 23:59 GMT
