Skip to content

W
wuxhqi
Open
Opened by Chiquita Yoo@chiquita36483
Report abuse New issue

If there's Intelligent Life out There


Optimizing LLMs to be great at specific tests backfires on Meta, Stability.

-. -. -. -. -. -. -

When you buy through links on our website, we might make an affiliate commission. Here's how it works.

Hugging Face has actually launched its 2nd LLM leaderboard to rank the best language models it has actually evaluated. The brand-new leaderboard seeks to be a more difficult uniform requirement for evaluating open large language design (LLM) performance across a variety of jobs. Alibaba's Qwen models appear dominant in the leaderboard's inaugural rankings, taking 3 spots in the top 10.

Pumped to announce the brand new open LLM leaderboard. We burned 300 H100 to re-run brand-new assessments like MMLU-pro for all major open LLMs!Some learning:- Qwen 72B is the king and Chinese open models are controling overall- Previous assessments have actually become too easy for current ... June 26, 2024

Hugging Face's 2nd leaderboard tests language designs across four jobs: understanding screening, reasoning on extremely long contexts, complex mathematics capabilities, and instruction following. Six criteria are utilized to evaluate these qualities, with tests consisting of fixing 1,000-word murder mysteries, explaining PhD-level concerns in layperson's terms, and a lot of challenging of all: high-school math formulas. A full breakdown of the criteria utilized can be found on Hugging Face's blog site.

The frontrunner of the new leaderboard is Qwen, Alibaba's LLM, which takes first, 3rd, and 10th place with its handful of variants. Also appearing are Llama3-70B, Meta's LLM, and a handful of smaller sized open-source projects that managed to surpass the pack. Notably absent is any indication of ChatGPT; Hugging Face's leaderboard does not check closed-source models to guarantee reproducibility of outcomes.

Tests to qualify on the leaderboard are run solely on Hugging Face's own computer systems, forum.altaycoins.com which according to CEO Clem Delangue's Twitter, setiathome.berkeley.edu are powered by 300 Nvidia H100 GPUs. Because of Hugging Face's open-source and collaborative nature, anyone is totally free to send brand-new models for testing and admission on the leaderboard, wiki.myamens.com with a new ballot system focusing on popular new entries for screening. The leaderboard can be filtered to show only a highlighted selection of substantial models to prevent a confusing glut of little LLMs.

As a pillar of the LLM space, Hugging Face has ended up being a relied on source for LLM learning and neighborhood collaboration. After its very first leaderboard was launched last year as a means to compare and reproduce screening outcomes from numerous recognized LLMs, the board rapidly took off in popularity. Getting high ranks on the board became the goal of lots of developers, little and large, and as models have actually ended up being typically more powerful, 'smarter,' and enhanced for the particular tests of the first leaderboard, its results have ended up being less and less significant, for this reason the production of a second version.

Some LLMs, including newer variations of Meta's Llama, badly underperformed in the brand-new leaderboard compared to their high marks in the first. This came from a pattern of over-training LLMs just on the very first leaderboard's standards, leading to regressing in real-world efficiency. This regression of performance, thanks to hyperspecific and self-referential information, follows a trend of AI performance growing even worse in time, proving once again as Google's AI responses have actually revealed that LLM performance is just as excellent as its training information and that real artificial "intelligence" is still many, numerous years away.

Remain on the Innovative: Get the Tom's Hardware Newsletter

Get Tom's Hardware's finest news and in-depth evaluations, straight to your inbox.

Dallin Grimm is a contributing writer for Tom's Hardware. He has actually been developing and breaking computers given that 2017, working as the resident child at Tom's. From APUs to RGB, Dallin guides all the current tech news.

Moore Threads GPUs presumably reveal 'exceptional' inference efficiency with DeepSeek designs

DeepSeek research study recommends Huawei's Ascend 910C provides 60% of Nvidia H100 reasoning performance

Asus and MSI hike RTX 5090 and forum.altaycoins.com RTX 5080 GPU prices by up to 18%

-. bit_user. is only as great as its training data which true artificial "intelligence" is still numerous, many years away. First, this declaration discount rates the role of network architecture.

The meaning of "intelligence" can not be whether something procedures details exactly like people do, or else the look for additional terrestrial intelligence would be totally futile. If there's intelligent life out there, it most likely does not think quite like we do. Machines that act and act smartly likewise need not necessarily do so, either. Reply

-. jp7189. I do not like the click-bait China vs. the world title. The truth is qwen is open source, open weights and can be run anywhere. It can (and has currently been) fine tuned to add/remove bias. I praise hugging face's work to create standardized tests for LLMs, and for putting the concentrate on open source, open weights first. Reply

-. jp7189. bit_user said:. First, this declaration discounts the function of network architecture.

Second, intelligence isn't a binary thing - it's more like a spectrum. There are numerous classes cognitive tasks and capabilities you may be acquainted with, asteroidsathome.net if you study kid development or animal intelligence.

The definition of "intelligence" can not be whether something procedures details exactly like people do, or archmageriseswiki.com else the search for additional terrestrial intelligence would be completely useless. If there's intelligent life out there, it most likely doesn't think quite like we do. Machines that act and act smartly also need not always do so, either. We're creating a tools to help people, therfore I would argue LLMs are more valuable if we grade them by human intelligence standards. Reply

- View All 3 Comments

Most Popular

Tomshardware is part of Future US Inc, a worldwide media group and leading digital publisher. Visit our business website.

- Terms.