+ Reply
Results 1 to 4 of 4

Your favorite Apple, iPhone, iPad, iOS, Jailbreak, and Cydia site.


Thread: Several Mobile Devices Makers Found Gaming Benchmarks

is a discussion within the

Mac News

forums, a part of the

General Apple/Mac

section;
After the recent report that Samsung artificially inflates its benchmarking scores, the well-respected hardware review site, AnandTech , published evidence that suggests nearly all mobile manufacturers with the exception of
...
  1. #1
    MMi Staff Writer Akshay Masand's Avatar
    Join Date
    Sep 2011
    Location
    New York City
    Posts
    3,789
    Thanks
    3
    Thanked 118 Times in 104 Posts

    Default Several Mobile Devices Makers Found Gaming Benchmarks


    After the recent report that Samsung artificially inflates its benchmarking scores, the well-respected hardware review site, AnandTech, published evidence that suggests nearly all mobile manufacturers with the exception of Apple and Motorola use CPU/GPU optimizations to game a variety of benchmark tests. Samsung and other OEMs use a variety of methods to enhance device performance when a benchmark is detected. An example of this was seen with the Galaxy S4 device where it raised its thermal limits (and max GPU frequency) to get an edge on certain benchmarks and also raised its CPU voltage/frequency to its highest state when a benchmark was sensed. This tactic was discovered being used with multiple manufacturers including LG, HTC and ASUS as well.

    AnandTech highlights devices that detect benchmarks and immediately respond with max CPU frequency. This can be seen in the table below:



    The folks over at AnandTech wrote the following about the matter:

    With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device that runs this silly CPU optimization. It’s possible that older Motorola devices might’ve done the same thing, but none of the newer devices we have on hand exhibited the behavior. It’s a systemic problem that seems to have surfaced over the last two years, and one that extends far beyond Samsung.
    The whole situation was mentioned as being a continual “cat and mouse” game discovering which devices have optimized for which benchmarks because targeted benchmarks must be avoided. The following was said regarding the matter:

    The only realistic solution is to continue to evolve the suite ahead of those optimizing for it. The more attention you draw to certain benchmarks, the more likely they are to be gamed. We constantly play this game of cat and mouse on the PC side, it's just more frustrating in mobile since there aren’t many good benchmarks to begin with. […]

    There's no single solution here, but rather a multi-faceted approach to make sure we’re ahead of the curve. We need to continue to rev our test suite to stay ahead of any aggressive OEM optimizations, we need to petition the OEMs to stop this madness, we need to work with the benchmark vendors to detect and disable optimizations as they happen and avoid benchmarks that are easily gamed.
    It should be noted that despite all of the effort that OEMs seem to be putting into benchmark optimizations, the gains seem to be negligible. The impact on CPU tests revealed a 0% to 5% performance increase and a less than 10% increase on GPU benchmarks.

    Those of you who are interested in reading more about the topic should hit the source link below. On that note, how do you feel about these manufacturers and what they are doing?

    Source: AnandTech

    Twitter: @AkshayMasand

  2. #2
    iPhone? More like MyPhone holyshnikes's Avatar
    Join Date
    Oct 2010
    Posts
    206
    Thanks
    1
    Thanked 27 Times in 20 Posts

    Ok, sure the tests don't boost performance that much, but it's the principal of the thing. Stand by your products if you think they're better. Don't try to fake better performance and try to convince consumers that they are better than they actually are.

  3. #3
    iPhone? More like MyPhone
    Join Date
    Jun 2012
    Posts
    229
    Thanks
    0
    Thanked 25 Times in 15 Posts

    Quote Originally Posted by holyshnikes View Post
    Ok, sure the tests don't boost performance that much, but it's the principal of the thing. Stand by your products if you think they're better. Don't try to fake better performance and try to convince consumers that they are better than they actually are.
    i think what you're referring to is marketing. every company markets and promises things the device isn't capable to do. I think there should be a curb on this kind of false advertising / marketing.

  4. #4
    iPhone? More like MyPhone ohthatguyagain's Avatar
    Join Date
    Sep 2008
    Location
    Texas
    Posts
    288
    Thanks
    46
    Thanked 14 Times in 9 Posts

    Quote Originally Posted by kalpesh78 View Post
    i think what you're referring to is marketing. every company markets and promises things the device isn't capable to do. I think there should be a curb on this kind of false advertising / marketing.
    Agreed. It's false advertising.

    I think the benchmark folks should band together and create a coalition with a seal of approval. If you want to be legitimately benchmarked and try for the gold seal, your device will be inspected. If, during the inspection, evidence is found with intent to defraud the test or falsify results, then the product should be publicly banned.

    The coalition would, of course, have to charge the company for the exam. However, the buck could be passed to consumers. I'd buy something that was certified faster, even if the gold seal meant the product cost and extra five bucks.

Posting Permissions
  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts