X Makes Its Algorithm Open Source Amid Transparency Fine & Grok Issues

The first time Twitter’s algorithm was largely open-sourced was in 2023. Elon Musk, the billionaire owner of Tesla, had just recently purchased the platform at the time, and he stated that he was on a mission to reorganise it in order to increase its transparency.

The main ranking system for the “For You” feed is covered in this release, which is available on GitHub at xai-org/x-algorithm.

The decision was made in response to intense public criticism and regulatory pressure on the platform.

Critics quickly criticised the algorithm’s code release for being “transparency theatre”, pointing out that it was “incomplete” and didn’t provide any insight into the organisation’s inner workings or the reasons behind the code’s functionality.

As promised by Musk last week, the website (now rebranded as X) has once more made its algorithm open source. In seven days, he had stated, “We will make the new algorithm, including all code used to determine what organic and advertising posts are recommended to users, open source.” Additionally, Musk pledged to make the algorithm transparent every four weeks going forward.

X posted an easy-to-read explanation of its feed-generating code and a schematic of the program’s operation on GitHub on Tuesday.

Although what has been disclosed isn’t especially revolutionary, it does offer a glimpse behind the algorithmic curtain. The figure illustrates how the site’s algorithm looks at recent in-network posts as well as the user’s engagement history, including the posts they have clicked on. Additionally, it analyses “out-of-network” posts, that is, content from accounts the user doesn’t necessarily follow, that it thinks the user could find interesting using machine learning.

The system then eliminates specific types of postings, such as those from restricted accounts, posts containing muted keywords, and content that has been judged to be too violent or spam-like. The material is then ranked by the algorithm according to what it believes the user will find most interesting. In order to prevent readers from seeing a large number of identical postings, this approach takes into account elements like content diversity and relevancy. The possibility that a user would like, comment on, repost, favourite, or otherwise interact with a piece of material is another factor that the algorithm takes into account.

X claims that the entire system is AI-based. The technology “relies entirely” on the company’s “Grok-based transformer” to “learn relevance from user engagement sequences,” according to the GitHub article published on Tuesday. To put it another way, Grok examines what you click on and like, then feeds that data into the recommendation algorithm. Additionally, the article points out that there is no “manual feature engineering for content relevance”, which means that people do not deliberately modify the algorithm’s determination of what is relevant. The automation “significantly reduces the complexity in our data pipelines and serving infrastructure,” the statement continues.

Why is X suddenly disclosing all of this? It’s not entirely evident. Musk has stated in the past that he wants the platform to be a model of corporate transparency, a sentiment that persists to this day. Musk stated that offering “code transparency” would be “incredibly embarrassing at first” but would eventually “lead to rapid improvement in recommendation quality” when the Twitter algorithm was initially made public in 2023. “Above all, we hope to earn your trust,” he continued. The platform announced a “new era of transparency” for Twitter with the release of its initial code.

Despite Musk’s claims of transparency, some areas of the platform may have become less transparent after Musk took charge. Twitter was famously compelled to change from being a public corporation to a private one when the tech billionaire acquired it in 2022; this change isn’t usually associated with transparency. Although the website used to publish several transparency reports annually, X didn’t publish its first one until September 2024. European Union regulators fined X $140 million in December for allegedly violating “transparency obligations” under the Digital Services Act (DSA). They also claimed that the site’s verification tick mark system made it more difficult for users to assess the legitimacy of specific accounts.

The methods that Grok, X’s chatbot, has been exploited to produce and disseminate sexualised content have also put the company under scrutiny over the past month. In recent weeks, the platform has come under scrutiny from both the California Attorney General’s office and congressional politicians due to allegations that Grok has been used to produce naked photos of women and juveniles. This call to openness may therefore be seen by others as just theatrical.

More technical details regarding the release are that 

For Grok-based architecture, the new algorithm predicts user interaction and ranks content using the same Transformer architecture as the Grok model.

It is also known that the monthly updates occur every four weeks, as Elon Musk promised to update the code and provide thorough developer notes outlining any modifications. Although the framework is available to the public, some have referred to it as “theatrical transparency” because crucial elements like model weights and training data are still hidden.