Difference between revisions of "Thread:User talk:Xor/General Ablation Study Framework"
(New thread: General Ablation Study Framework) |
m |
||
| (2 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
| − | To accurately measure the performance of different components of a bot, we do ablation study by disabling them. However, due to the popularity of hand-made bot frameworks, it is generally hard to compare your bot to other bots component-wise, even when source code is given. | + | To accurately measure the performance of different components of a bot, we do ablation study by disabling each of them. However, due to the popularity of hand-made bot frameworks, it is generally hard to compare your bot to other bots component-wise, even when source code is given. |
Since we cannot have everyone rebase onto a common framework, another approach would be replacing part of the behavior of a robot dynamically. This should also work with bots without source code. | Since we cannot have everyone rebase onto a common framework, another approach would be replacing part of the behavior of a robot dynamically. This should also work with bots without source code. | ||
| − | A robot can be divided into 3 basic components: Movement, Gun and Radar. Replacing behavior from one component is as simple as hooking calls like setBodyXXX, setGunXXX, setRadarXXX to IAdvancedRobotPeer. This way we can easily compare different guns from different bots while having movements fixed | + | A robot can be divided into 3 basic components: Movement, Gun and Radar. Replacing behavior from one component is as simple as hooking calls like setBodyXXX, setGunXXX, setRadarXXX to IAdvancedRobotPeer. This way we can e.g. easily compare different guns from different bots while having movements fixed, even without doing anything with the code. |
There are certainly drawbacks: | There are certainly drawbacks: | ||
| − | + | # Replaced components are still consuming CPU while contributing no behavior. | |
| − | + | # No fine-grained way of comparing sub-components, e.g. Bullet Shadow | |
| − | + | # Certain features may not work properly without manually changing code, e.g. bullet shadow may have wrong gun firing angles | |
Also, frameworks like this should only be used for experimental purposes, and forbidden in competitions unless given permissions from original authors. | Also, frameworks like this should only be used for experimental purposes, and forbidden in competitions unless given permissions from original authors. | ||
Latest revision as of 10:44, 19 February 2023
To accurately measure the performance of different components of a bot, we do ablation study by disabling each of them. However, due to the popularity of hand-made bot frameworks, it is generally hard to compare your bot to other bots component-wise, even when source code is given.
Since we cannot have everyone rebase onto a common framework, another approach would be replacing part of the behavior of a robot dynamically. This should also work with bots without source code.
A robot can be divided into 3 basic components: Movement, Gun and Radar. Replacing behavior from one component is as simple as hooking calls like setBodyXXX, setGunXXX, setRadarXXX to IAdvancedRobotPeer. This way we can e.g. easily compare different guns from different bots while having movements fixed, even without doing anything with the code.
There are certainly drawbacks:
- Replaced components are still consuming CPU while contributing no behavior.
- No fine-grained way of comparing sub-components, e.g. Bullet Shadow
- Certain features may not work properly without manually changing code, e.g. bullet shadow may have wrong gun firing angles
Also, frameworks like this should only be used for experimental purposes, and forbidden in competitions unless given permissions from original authors.
